Skip to content

feat: Implement two-sided verification check with check modes#487

Merged
MikaelMayer merged 203 commits intomainfrom
feat/two-sided-verification-check
Mar 12, 2026
Merged

feat: Implement two-sided verification check with check modes#487
MikaelMayer merged 203 commits intomainfrom
feat/two-sided-verification-check

Conversation

@MikaelMayer
Copy link
Contributor

@MikaelMayer MikaelMayer commented Feb 26, 2026

Summary

Replaces the single-sided reachCheck flag with a two-sided verification framework using orthogonal check mode and check amount flags. Each proof obligation now produces a VCOutcome with independent satisfiability and validity properties, enabling richer diagnostic feedback.

Problem

We want to perform richer checks on assert statements beyond simple validity. Covers are existential checks where forking into two means the results are linked by an OR, so they are not suitable for detecting assertions that surely fail along a path. To find such failures, checks must be encoded as assertions, and we need extended diagnostics for them.

A previous PR opened the way by adding a reachability check, demonstrating that two checks per command are feasible. However, the reachability check missed an important case for bug-finding mode: from reachability + validity alone, we cannot derive the result of reachability + satisfiability. By testing both P ∧ Q (satisfiability) and P ∧ ¬Q (validity) where P is the path condition and Q is the property, we get two checks that together determine the validity and satisfiability of Q given P, and also derive reachability.

Solution

Two orthogonal flags replace reachCheck:

  • Check Mode (--check-mode): deductive (default) or bugFinding
  • Check Amount (--check-amount): minimal (default) or full

A per-statement @[fullCheck] annotation can override the global check amount.

Possible outcomes by mode

Default mode (deductive, minimal): validity check only for asserts, satisfiability check only for covers.

For assert statements (validity only, satisfiability masked to unknown):

Emoji Label Meaning
✔️ always true if reachable Validity passed, property always true if reachable
can be false and is reachable Validity failed, solver found a reachable counterexample (with model)
unknown Solver could not determine validity

For cover statements (satisfiability only, validity masked to unknown):

Emoji Label Meaning
can be true and is reachable Satisfiability passed, property can be true and is reachable from declaration entry
✖️ refuted if reachable Satisfiability failed, property always false if reachable
unknown Solver could not determine satisfiability

Bug-finding mode (bugFinding, minimal): satisfiability check only for all statement types. Same as the cover table above.

Full mode (full): both checks run, all 9 outcomes possible. The last two columns show the error reporting level in SARIF output for each mode (✅ = pass, 🔴 = error, 🟡 = warning, 🔵 = note).

Emoji Label P ∧ Q P ∧ ¬Q Inferred reachability Meaning Deductive mode error level Bug finding mode error level
always true and is reachable sat unsat Property always true, reachable from declaration entry
always false and is reachable unsat sat Property always false, reachable from declaration entry 🔴 🔴
🔶 indecisive and reachable sat sat Reachable from declaration entry, solver found models for both the property and its negation 🔴 🔵
⛔/❌ unreachable unsat unsat Dead code, path unreachable (⛔ warning for assert, ❌ error for cover) 🟡/🔴 🟡/🔴
can be true and is reachable sat unknown Property can be true and is reachable from declaration entry, validity unknown 🔴 🔵
✖️ refuted if reachable unsat unknown Property always false if reachable, reachability unknown 🔴 🔴
can be false and is reachable unknown sat Solver found a model for P ∧ ¬Q: path is reachable from declaration entry and Q can be false, but satisfiability of Q is unknown 🔴 🔵
✔️ always true if reachable unknown unsat Property always true if reachable, reachability unknown
unknown unknown unknown Both checks inconclusive 🔴 🔵

Testing

All existing tests updated. New tests cover the full outcome matrix including per-statement @[fullCheck] annotations. BoogieToStrata integration tests, Python analysis tests, and SARIF output all updated.

Implement the two-sided verification check design that distinguishes between
'always true', 'always false', 'indecisive', and 'unreachable' outcomes.

Key changes:
- Add checkSatAssuming to SMT Solver for assumption-based queries
- Replace Outcome inductive with VCOutcome structure containing two SMT.Result fields
- Add CheckMode enum (full/validity/satisfiability) to Options
- Update encoder to emit two check-sat-assuming commands
- Update SARIF output to handle nine possible outcome combinations
- Default to validity mode for backward compatibility

The two-sided check asks:
1. Can the property be true? (satisfiability check)
2. Can the property be false? (validity check)

This enables distinguishing:
- pass (sat, unsat): always true and reachable
- refuted (unsat, sat): always false and reachable
- indecisive (sat, sat): true or false depending on inputs
- unreachable (unsat, unsat): path condition contradictory
- Five partial outcomes when one check returns unknown

Breaking change: VCResult API changed, all consumers must be updated.
Tests need updating to reflect new default behavior (validity mode only).

See TWO_SIDED_CHECK_IMPLEMENTATION.md for complete implementation details.
- Add CLI parsing for --check-mode flag (full/validity/satisfiability)
- Remove deprecated --reach-check flag
- Update help message with check mode documentation
- Fix StrataVerify to use 'outcome' field instead of 'result'
- Update emoji symbols for better visual distinction:
  - ✅ for pass (valid and reachable)
  - ✔️ for always true if reachable
  - ✖️ for refuted if reachable
  - ❌ for refuted (always false and reachable)
  - ⛔ for unreachable
  - 🔶 for indecisive
  - ➕ for satisfiable
  - ➖ for reachable and can be false
- Add metadata fields: fullCheck, validityCheck, satisfiabilityCheck
- Add helper methods to check for these annotations
- Update verifySingleEnv to check metadata before using global checkMode
- Annotations override global --check-mode flag for specific statements
- Add VCOutcomeTests.lean with all 9 outcome combinations
- Test both predicate methods and emoji/label rendering
- Use named arguments for clarity
- Update SMTEncoderTests to use full check mode for existing tests
- Ensures backward compatibility with expected 'pass' outcome
- Add VCOutcomeTests.lean with all 9 outcome combinations
- Each test shows emoji and label in output for easy verification
- Use named arguments for clarity
- Update SMTEncoderTests to use full check mode for existing tests
- Ensures backward compatibility with expected 'pass' outcome
- Add VCOutcomeTests.lean with all 9 outcome combinations
- Use formatOutcome helper to avoid repetition
- Each test shows emoji and label in output
- Use named arguments for clarity
- Update SMTEncoderTests to use full check mode
- Ensures backward compatibility with expected 'pass' outcome
- Document CLI flag integration
- Document per-statement annotations
- Document emoji updates
- Document comprehensive test suite
- Document test fixes for backward compatibility
- Fix StrataVerify to properly format Except String VCOutcome
- Update StrataMain to use vcResult.outcome instead of vcResult.result
- Use isRefuted/isRefutedIfReachable predicates for failure detection
- Format outcomes with emoji and label
Clarifies that refuted outcome means reachable and always false
…ters

- Rename isRefuted -> isRefutedAndReachable
- Rename isIndecisive -> isIndecisiveAndReachable
- Rename isRefutedIfReachable -> isAlwaysFalseIfReachable
- Add backward compatibility aliases
- Add cross-cutting predicates: isAlwaysFalse, isAlwaysTrue, isReachable
- Enables filtering outcomes by properties across multiple cases
…ariants

- isPass: true if validityProperty is unsat (always true), regardless of reachability
- isPassAndReachable: true if (sat, unsat) - proven reachable and always true
- isPassIfReachable: true if (unknown, unsat) - always true if reachable
- Update label/emoji to use isPassAndReachable and isPassIfReachable
- Update test comments to reflect new naming
- Add backward compatibility alias isAlwaysTrueIfReachable
…overs all sat cases

- isSatisfiable: true for any sat satisfiabilityProperty
- isSatisfiableValidityUnknown: specific case (sat, unknown)
- Rename isPassIfReachable -> isPassReachabilityUnknown
- Rename isAlwaysFalseIfReachable -> isAlwaysFalseReachabilityUnknown
- Rename isReachableAndCanBeFalse -> isCanBeFalseAndReachable
- All predicates now have reachability info at the end
- Add backward compatibility aliases for all old names
- Nine base cases without 'is': passAndReachable, refutedAndReachable, etc.
- Derived predicates with 'is': isPass, isSatisfiable, isReachable, etc.
- Base cases represent exact outcome combinations
- Derived predicates check properties across multiple outcomes
- Update SarifOutput to use base cases in outcomeToLevel/outcomeToMessage
- Update label/emoji functions to use base cases
- Maintain backward compatibility aliases for all old names
- Add VerificationMode enum: deductive vs bugFinding
- Deductive mode: only pass is success, anything not proven is error/warning
- Bug finding mode: refuted is error, unknown is acceptable warning
- Group outcomes by severity (one .none, one .error, one .warning, one .note per mode)
- Default to deductive mode for backward compatibility
…e isAlwaysFalse

- Deductive mode: only pass/unreachable are success/note, everything else is error
- Bug finding mode: use isAlwaysFalse predicate instead of listing base cases
- Cleaner and more maintainable
…achable is warning in deductive

- Consistent naming: use 'alwaysFalse' instead of 'refuted' in base cases
- Deductive mode: unreachable is warning (dead code detection)
- Update all references in Verifier.lean and SarifOutput.lean
- Maintain backward compatibility aliases
- Replace isAlwaysFalse with explicit base cases: alwaysFalseAndReachable, alwaysFalseReachabilityUnknown
- Add comment listing all error cases in deductive mode
- Clearer mapping from base cases to severity levels
- Remove 'Verification succeeded/failed' language
- Use neutral descriptions: 'Always true and reachable', 'Always false and reachable'
- Messages work for any property type (assertion, invariant, requires, etc.)
- Shorter and clearer messages
…nknown outcomes

- alwaysFalseReachabilityUnknown has validityProperty = unknown (not sat), no counterexample
- unknown outcome can have models from either satisfiability or validity property
- Show models from both properties when available for unknown outcome
- alwaysFalseReachabilityUnknown has validityProperty = unknown (no model)
- unknown outcome also has no models (Result.unknown carries no data)
- Only Result.sat carries counterexample models
…rties

- Eliminates redundant predicate checks in outcomeToMessage
- Single exhaustive match covers all 9 base cases plus error cases
- More concise and easier to verify correctness
- Test predicates, messages, and severity levels for each outcome
- Verify deductive and bug finding modes produce correct SARIF levels
- Self-contained test outputs with no numbered comments
- Tests ensure SARIF output matches predicate semantics
- Add missing validityCheck parameter (now takes satisfiabilityCheck and validityCheck)
- Use Except.ok/Except.error to avoid ambiguity
@MikaelMayer MikaelMayer enabled auto-merge March 11, 2026 15:17
@aqjune-aws
Copy link
Contributor

I will review this by tomorrow morning.

@keyboardDrummer
Copy link
Contributor

incomplete, then indeed you don't care about that case.
However, if we think providing complete preconditions is simpler than providing a correctness proof, then I think "test for all proven bugs assuming that the preconditions are complete" could be useful. There are many bugs that only trigger on some inputs, so we could find a lot more bugs with this mode than with the bug finding mode that you're describing.

I understand better now, thank you very much for the explanation. It seems rational to have an intermediate between bug finding (where preconditions are typically absent) and deductive (where complete preconditions are expected), where we want bug checking in full mode to report (sat, sat) as a failure.

I thought about it a lot and decided to make it a mode (not an additional flag), but that mode forces the check level to be full, as we can't report negative counter examples if we don't check validity.

ok, thanks!

- Remove unused _md from Imperative.SMT.dischargeObligation
- Refactor outcomeToLevel to use cleaner if-else structure
- Remove backward compatibility aliases not in main
- Fix outdated comment in maskOutcome
- Use _ instead of _satResult in Verify.lean
… sat, val)

Addresses reviewer comment to use direct pattern matching instead of
nested boolean logic. Each case is now a single match arm.
joscoh
joscoh previously approved these changes Mar 12, 2026
@MikaelMayer MikaelMayer disabled auto-merge March 12, 2026 18:30
@MikaelMayer MikaelMayer enabled auto-merge March 12, 2026 18:34
@MikaelMayer MikaelMayer added this pull request to the merge queue Mar 12, 2026
Merged via the queue into main with commit cd0a13f Mar 12, 2026
15 checks passed
@MikaelMayer MikaelMayer deleted the feat/two-sided-verification-check branch March 12, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants